Integrated vision-based and inertial sensor systems for use in vehicle navigation

ABSTRACT

A navigation system useful for providing speed and heading and other navigational data to a drive system of a moving body, e.g., a vehicle body or a mobile robot, to navigate through a space. The navigation system integrates an inertial navigation system, e.g., a unit or system based on an inertial measurement unit (IMU). with a vision-based navigation system unit or system such that the inertial navigation system can provide real time navigation data and the vision-based navigation can provide periodic, but more accurate, navigation data that is used to correct the inertial navigation system&#39;s output. The navigation system was designed with the goal in mind of providing low effort integration of inertial and video data. The methods and devices used in the new navigation system address problems associated with high accuracy dead reckoning systems (such as a typical vision-based navigation system) and enhance performance with low cost IMUs.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.15/919,842, filed Mar. 13, 2018, which claims the benefit of U.S.Provisional Application No. 62/471,200, filed Mar. 14, 2017, which areboth incorporated herein by reference in their entirety.

BACKGROUND 1. Field of the Invention

The present description generally relates to vision-based navigation (orpositioning) systems for use in moving bodies such as automobiles andother vehicles, mobile robots, handheld devices, and the like. Moreparticularly, the present description is directed toward a navigationsystem for a moving body (such as a vehicle) that combines outputs of acomputer vision-based navigation system (or also called a videonavigation system) to effectively estimate a current speed (or velocity)and/or heading of a vehicle or other moving body.

2. Relevant Background

It is becoming increasingly common for vehicles to include vision-basedpositioning or navigation systems that are used to facilitate driverlessor automated navigation of the vehicle and to provide assistance to ahuman pilot or driver of the vehicle. For example, a vision-basednavigation system may be provided on an automobile (or other vehiclesuch as an unmanned aerial vehicle (UAV) or the like), and it mayinclude one or more cameras that are used to capture image frames thatare processed to provide estimates of the vehicle's current speed andheading as well as the vehicle's current position. The outputs of thevision-based navigation system can be provided to the driver to assistin navigation or provided to a driver system to allow an automatedvehicle to navigate through traffic and/or to avoid obstacles as itmoves toward a defined destination.

For vision-based navigation systems to be more widely adopted invehicles, there are a number of operational challenges that need to beaddressed. Computer vision-based positioning system are capable ofmeasuring three translational dimensions as well as three rotationaldimensions that allow it to provide useful navigation informationincluding a vehicle's position, speed, and heading. However, onelimitation of vision-based positioning systems is the computationaleffort required to calculate results including the vehicle's currentspeed. Even with desktop-type hardware, it can be difficult orimpossible to run a vision-based positioning system in close to realtime, which is demanded or at least desired for navigation of a vehicle.

Another problem associated with vision-based navigation systems isinformation loss. Rolling shutter and global shutter are the twofundamental architectures for image sensors (or cameras) used for imagecapture in a vision-based navigation system. A global shutter sensorexposes an entire frame at once and then records it or reads it out. Arolling shutter sensor staggers the start and stop times for theexposure on a row-by-row basis along the sensor, which results in animage where each row is captured at a slightly different time. A benefitof the rolling shutter is that it requires less semiconductor area toimplement and is, thus, less expensive than the global shutterarchitecture.

However, a challenge with use of rolling shutter sensors is that theimage data lacks spatial coherence across rows when there is motion ineither the scene or the camera (or both are moving). For example, arolling shutter sensor that is stationary may be used to capture animage of a rolling train (or other vehicle). Since each row is capturedat a different time and the train is in a different location for eachrow, the captured image gives the appearance that the train is sheared(leaning away from the direction of travel rather than being vertical).In another example, a moving camera may be used to capture an image ofthe surrounding space that may include a city's skyline. In this case,the camera is in a different place as each row is captured such thatskyline will appear slanted. In yet another example, a relativelystationary camera may be used to capture images of a spinning airplanepropeller. In this case, the rotational speed of the propeller is fastenough relative to the row-by-row capture rate that the propeller bladesin the captured image show significant distortion, and, in someinstances, the distortion is so great that the blades appeardisconnected from the hub. As a result, the impact on computervision-based navigation algorithms of rolling shutter information losscan be devastating. Specifically, for vision-based estimations inmotion, the algorithms need to understand the geometric relationshipsbetween different parts of the image, and this information can bedestroyed when the perspective changes between rows.

Hence, there remains a need for improvements to vision-based navigationsystems to address the associated problems including how to providenavigational data, such as the current speed, heading, location, and thelike, for a moving body such as an automobile on an ongoing or nearreal-time basis. Preferably, the new vision-based navigation systemswill be practical for the many existing and planned uses of vision-basednavigation systems and will have a relatively low implementation cost(e.g., rather than providing an implementation that is optimized formaximum performance but is impractical and/or very expensive).

SUMMARY

Briefly, the inventors designed a navigation system for providing speedand heading and other navigational data useful by a drive system of amoving body (e.g., a body of a vehicle, a mobile robot, a handhelddevice, or the like) to navigate through a space. The navigation systemintegrates an inertial navigation system (e.g., a unit or system basedon an inertial measurement unit (IMU)) with a vision-based navigationsystem (or video navigation (VN) unit or system) such that the inertialnavigation system can provide real time navigation data and thevision-based navigation can provide periodic—but moreaccurate—navigation data that is used to correct the inertial navigationsystem's output.

The navigation system was designed with the goal in mind of providinglow effort integration of inertial and video data. The methods anddevices used in the new navigation system address problems associatedwith high accuracy dead reckoning systems (such as a typical VNunit/system) and provide better performance from low cost IMUs. Theinventors recognized, with regard to problems with present navigationtechniques, inertial and video navigation would benefit from aiding bythe other, but the complexities of video processing make measurementvery processor intensive. Inertial navigation units drift cubic orquadratic with time. Video navigation can give drift characteristicsthat are linear with distance traveled, but real-time video navigationon low-cost processors is not currently practical.

The navigation systems described herein use new techniques and devicesthat were designed with the recognition that video navigation (VN) worksby measuring differences between two frame pairs. The calculations cantake many multiples of real time (e.g., a predefined time periodconsidered to provide “real-time” output) to complete. One unique ideapresented by the inventors involves taking a quick frame pair at typicalvideo rates (e.g., 1/30 second between frames in pair) and beginprocessing with a VN unit or system. While the VN system is processingthe frame pair, the inertial navigation unit or system (e.g., a modulewith one or more IMUS) propagates its navigation data or measurements(e.g., estimated speed and heading of a moving body such as a vehicle orits body). Over the course of this processing interval, accelerationmeasurement errors for the inertial navigation unit will startaccumulating. At the end of the processing interval, the VN systemreturns an accurate measurement of camera displacement and rotationbetween those two frames of the captured frame pair (i.e., itsnavigation data or output over a fixed time period giving velocity andangular rate measurements). The VN system's measurements are then usedby the navigation system to retroactively discipline the inertialnavigation unit (or its IMU(s)), and another frame pair is captured bythe VN system for processing during the next processing period with theinertial navigation unit working from the accurate data from the VNsystem until this next processing period ends.

More particularly, a system is described for use in navigating a vehicleor other moving body (e.g., a UAV, a mobile robot, a handheld device, orthe like). The system includes an imaging unit, on a moving body,providing a first source of heading and speed estimates for the movingbody based on positions of features identified in images obtained atdifferent times. Also, on the moving body, the system includes aninertial navigation unit providing a second source of heading and speedestimates for the moving body based on at least one of gyroscopicmeasurements and accelerometer measurements. Further, the systemincludes a filter combining the heading and speed estimates from thefirst and second sources to provide an overall system heading and speedestimate for the moving body. In operation, the filter uses the headingand speed estimates from the imaging unit to bound drifts in the headingand speed estimates provided by the inertial navigation unit.

In some embodiments of the system, the filter sends the heading andspeed estimates provided by the inertial navigation unit to the imagingunit to increase a speed of calculating the heading and speed estimatesbased on the positions of the features identified in images obtained atdifferent times. The navigation system may be configured such that theimaging unit provides the heading and speed estimates based on thepositions of features identified in images obtained at regular timeintervals. In the same or other implementations of the system, theimaging unit provides the heading and speed estimates based on thepositions of features identified in images obtained at time intervalsdetermined by the filter based on performance of the inertial navigationunit. In some cases, the imaging unit provides the heading and speedestimates based on the positions of features identified in imagesobtained at regular intervals of distance travelled. Additionally, itmay be useful for the imaging unit to be configured to provide theheading and speed estimates at a frequency based on computingconstraints.

In another exemplary embodiment, a navigation system is provided forvehicles and other moving bodies, and the navigation system ispositioned or mounted on a navigable body with a drive system. Duringoperations, the navigation system generates navigation data for use bythe drive system to move the navigable body through a space. Thenavigation system includes an inertial navigation unit including aninertial measurement unit (IMU) and an IMU dead reckoning moduleprocessing a set of measurements provided by the IMU to generate a firstoutput including a first velocity of the navigable body. The navigationsystem further includes a vision-based navigation system including animage sensor sequentially capturing a pair of images of the space andfurther including a vision-based dead reckoning module processing thepair of images to generate a second output including a second velocityof the navigable body. Additionally, the navigation system includes aprocessor executing code or software to provide an integration modulethat processes the first and second outputs to generate the navigationdata. The navigation data includes the first velocity or a firstposition calculated based on the first velocity during a time periodwhile the vision-based navigation system is generating the second outputat the end of which the first velocity is modified based on the secondvelocity.

In some embodiments of the navigation system, the navigation datafurther comprises a heading of the navigable body in the space. In theseembodiments, the heading may be provided in the first output of the IMU,and the vision-based dead reckoning module processes the pair of imagesbased on the heading to process portions of the pair of images withinformation predicted to be lost with further travel of the navigablebody or to predict portions of later frames with new information.

In the same or other embodiments, after the first velocity is modifiedbased on the second velocity, the IMU dead reckoning module processes anadditional set of measurements provided by the IMU to generate a thirdoutput including a third velocity of the navigable body. The integrationmodule can then process the third output to generate the navigation databased on the third output and the modified first velocity.

In some specific implementations, the time period (in which the outputfrom the IMU is relied upon) is in the range of 6 to 10 times a framerate of the image sensor. It may be useful for the image sensor toinclude at least two video cameras that are directed or aimed at anangular offset of 180 degrees. The image sensor may take the form of arolling shutter image sensor, and the vision-based navigation system mayfurther include a rolling shutter compensation module processing thepair of images based on a speed of the navigable body to reversedistortion caused by a rolling shutter effect. In most cases, the imagesensor is configured such that the pair of images include a region ofoverlap.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram of a moving body (such as a vehiclebody, a mobile robot, or handheld device moving through a space) with anavigation system of the present description;

FIG. 2 illustrates a graph showing an IMU position track at two samplerates;

FIG. 3 provides a graph of a static case/use of a single axisaccelerometer;

FIG. 4 is a graph or plot of accelerometer values integrated intovelocity values;

FIG. 5 is a graph or plot of results of integration of the accelerometeroutputs into position values;

FIG. 6 is a graph or plot of x acceleration in a circular test case ofan accelerometer of an IMU;

FIG. 7 is a graph or plot showing accelerometer error for the case ofFIG. 6 ;

FIG. 8 is a graph or plot of results of integration of accelerometeroutputs of FIG. 6 into velocity;

FIG. 9 is a plot or graph of velocity of error in the circular testcase;

FIG. 10 is a graph showing the full 2D position track for the circulartest case;

FIG. 11 is a graph of the error from truth over time for the track ofFIG. 10 ;

FIG. 12 illustrates two sequentially captured images from a camera(e.g., a frame pair from a camera(s) of a vision-based positioningsystem;

FIGS. 13 and 14 are graphs showing full 2D position tracking with whencorrection of inertial drift is provided via a CV system for the staticand circular motion simulations, respectively;

FIGS. 15A and 15B illustrate with two timelines the timing ofvision-based positioning system updates to an IMU system;

FIGS. 16A and 16B illustrate two sequentially captured images forprocessing by a vision-based positioning system (e.g., a frame pair)showing IMU heading data used to direct order of vision-based deadreckoning processing of the frame pair;

FIGS. 17A and 17B illustrate another frame pair captured by avision-based positioning system and use of IMU data to process the videodata; and

FIG. 18 illustrates a set of captured images/frames and use ofIMU-provided navigation data by the vision-based positioning system topredict where new data will appear in later-captured images/frames.

DETAILED DESCRIPTION

The present description provides improved devices and methods for use innavigation systems such as those provided in automobiles and othervehicles. More specifically, new navigation systems are taught thatintegrate inertial positioning devices and methods (e.g., positioningdata provided by an inertial measurement unit (IMU)) with the moreaccurate computing intensive computer vision-based positioning systemsand methods to provide navigation data in near real time to a vehicle'sdrive system.

FIG. 1 illustrates schematically a body 110 that is moving as shown witharrows 115 through a space. The body 110 may be nearly any moving bodysuch as a vehicle body, a body of a mobile robot, a handheld device, andthe like. The body 110 includes a drive system 120 operating to move 115the body 110 based on a set of navigation data 131 (e.g., currentposition in the space, overall heading and speed/velocity estimates, andthe like), and a speedometer 124 may be included to measure the speed ofthe body 110 and to provide its measurements to a rolling shuttercompensation module 142 (as explained below).

A navigation system 130, which may be included on the moving (ormovable) body 110 or offboard with wireless communications to thecomponents on the body, generates the navigation data 131 in near realtime to facilitate accurate navigation and/or control of the drivesystem 120. The navigation system 130 includes a processor 140 thatexecutes code or runs software to provide functionality of anintegration module 144 with a filter (or filtering mechanism) 148 thatprocesses and combines output (e.g., position, heading and speedestimates, and the like) 151 from an inertial navigation unit 150 andoutput (e.g., heading and speed estimates and the like) 171 from animaging unit (or vision-based positioning/navigation system) 170. Theprocessor 140 may also execute code or run software to providefunctionality of a rolling shutter compensation module 142 that may beused to process images 182 (or other images captured by cameras 176) tocorrect distortion. The various processors of the system are shownseparately for the sake of clarity of description, but it is understoodthat their functionality may be combined or redistributed among actualhardware components according to the needs of a particularimplementation.

The inertial navigation unit 150 includes a processor 152 (or may useprocessor 140) to execute code or software to provide the functionalityof a dead reckoning module 154 and to manage memory/data storagedevice(s) 160. The inertial navigation unit 150 is shown to include anIMU 156 that may include one or more accelerometers 157 measuringaccelerations 162 of the body 110. The IMU 156 may further include oneor more gyroscopes 158 measuring angular rates 163 of the body 110during movements 115. These IMU outputs 162 and 163 are processed by thedead reckoning module 154 to generate a dead-reckoned result 164 thatmay include a velocity 166 of the body 110 and a position 168 in thespace of the body 110, and the result 164 may further include headingand speed estimates that can be provided in the output 151 to theintegration module 144.

The imaging unit (or vision-based positioning/navigation system) 170also includes a processor 172 (or may use processor 140) to run softwareor execute code to provide the functionality of a dead reckoning module174 and manage memory/data storage device(s) 180. The imaging unit 170includes two or more cameras 176 (e.g., digital video cameras)positioned as discussed below on the body and operated to capture imagesexternal to the body 110, and these images include a frame pair 182 thatis chosen or identified by the module 174 periodically (e.g., at capturetime interval 183 that may match rate at which module 174 can computeits results). In brief, the vision unit 170 with its dead reckoningmodule 174 processes the frame pair 182 so as to identify one or morefeatures 184 in the images 182, and each of these features 184 has aparticular position 186 in each of the frames 182. A dead-reckonedresult 190 is computed by comparing these identified features 184 ineach frame of the pair 182 to determine a pose rate/velocity 192. Thisresult 192 along with other data such as heading and speed estimates isprovided as output as shown at 171 of the imaging unit 170.

The integration module 144 integrates the outputs 141 and 171 such aswith use of filter or filtering mechanism 148 to generate navigationdata 131 that is provided to the drive system 120, and this data 131 mayutilize position and other data 151 from the inertial navigation unit150 while the imaging unit 170 is processing a next frame pair 182 toprovide more accurate velocity and/or position data 171 that is used tocorrect for accumulating error or drift in the inertial navigation unit150. Then, with this correction, the inertial navigation unit 150 againprovides the navigation data 131 until a next frame pair 182 isprocessed to provide a new dead-reckoned result 190 with its moreaccurate pose rate or velocity 192 that is provided in the navigationdata 131 to the drive system 120.

With the navigation system 130 generally understood, it may now beuseful to describe in more detail exemplary implementations of thesystem 130 and its components such as the integration module 144, theinertial navigation unit 150, and the imaging unit 170. In someembodiments, the integration module 144 with its filter 148 provides loweffort vision-based correction of the IMU 156. Hence, the integrationmodule 144 and its underlying algorithms provide a novel method forintegrating a vision system and an IMU in such a way as to accommodatethe unique challenges of each technique of navigation while exploitingthe benefits of each. One goal for the navigation system 130 is todevelop a practical and lower cost implementation rather than onerigorously optimized for maximum performance.

The properties of an IMU are well understood. An IMU typically containssome number of accelerometers and some number of gyroscopes formeasuring rotation. The accelerometers, even in a relatively lowperformance IMU, are extremely accurate. The limitation of an IMU liesin the fact that in order to determine a position, the accelerations arefirst integrated into a velocity and then integrated again into aposition. This results in a position error that will grow proportionalto the square of time. Once errors in heading are included in thesolution (or output of the inertial navigation unit), the error growthrate can become closer to a function of time cubed. It is common toperform zero velocity updates (or ZUPTs) to limit the growth of thedrift.

The properties of a computer vision-based positioning system (or CVsystem) are less familiar and are still being characterized to someextent. A CV system is capable of measuring three translationaldimensions and three rotational, with much of the emphasis in thefollowing discussion being the translation measurements of the CVsystem. CV system measurements will be somewhat noisier that IMUmeasurements in the short term, but these measurements have the uniquebenefit that they are direct measurements of displacement and, thus,will not accumulate cubicly or quadratically in time. In an ideal case,the drift of a CV system can be constrained to be proportional todistance traveled; however, in one implementation described herein,drift will be more closely proportional to time. A limitation of CVsystems is the computational effort required to calculate results, aseven on desktop hardware it can be difficult or impossible to run closeto real time.

The navigation systems described herein use the excellent short termperformance of the IMU to dead reckon positions while the CV system iscalculating. The direct measurement of velocity by the CV system is thenused to do a known velocity update (KUPT) in a manner similar to a ZUPT,which serves to constrain the IMU drift. Thus, the navigation system'sdrift is controlled while the CV system's computational requirements arerelaxed.

As discussed with regard to FIG. 1 , the inertial navigation unit 150includes a dead reckoning module 154 that produces dead-reckoned results164 by processing outputs of the IMU 156. The IMU 156 outputsaccelerations 157 and angular rates 158. In a general 6 degree offreedom (6DOF) IMU, there will be acceleration values for forward,lateral, and up and angular rates for pitch, roll, and yaw. In order toreserve (x,y,z) for the world coordinates, this discussion will use(u,v,w) for the local IMU coordinate system and (φ,θ,ψ) for pitch, roll,and yaw. φ is rotation about the u axis, θ is rotation about the v axis,and ψ is rotation about thew axis. Assuming the IMU is mounted on avehicle body or other mobile body, the path can be constrained to twodimensions of translation, (x,y) or (u,v) depending, and one dimensionof rotation, ψ.

For the test case where the sensor is static, the true motion componentsare simple as they are all zero:s _(x)(t)=0 s _(y)(t)=0v _(x)(t)=0 v _(y)(t)=0a _(x)(t)=0 a _(y)(t)=0

For the test case where the sensor is traveling in a circular path, theradius of the circle is r and the angular rate is ω. To ensure thevehicle or other mobile body always faces the direction of travel,ψ=ωt+π/2. The true position, s, at any time is:s _(x)(t)=r cos(ωt)s _(y)(t)=r sin(ωt)Calculus gives the true velocity and acceleration:

${v_{x}(t)} = {\frac{ds_{x}}{dt} = {{- \omega}r{\sin( {\omega\; t} )}}}$${v_{y}(t)} = {\frac{ds_{y}}{dt} = {\omega\; r\;{\cos( {\omega t} )}}}$${a_{x}(t)} = {\frac{dv_{x}}{dt} = {{- \omega^{2}}r{\cos( {\omega\; t} )}}}$${a_{y}(t)} = {\frac{dv_{t}}{dt} = {{- \omega^{2}}r{\sin( {\omega t} )}}}$The circular motion gives sinusoidal components in world space, but, inpolar coordinates, all three sensors would ideally emit constant values:

${v_{mag}(t)} = {\sqrt{{v_{x}(t)}^{2} + {v_{y}(t)}^{2}} = \ {\omega\; r}}$${a_{mag}(t)} = {\sqrt{{a_{x}(t)}^{2} + {a_{y}(t)}^{2}} = \ {\omega^{2}r}}$$\begin{matrix}{{v_{ang}(t)} = {\tan^{- 1}( \frac{v_{x}(t)}{v_{y}(t)} )}} \\{= {\tan^{- 1}( {- \frac{\cos\mspace{11mu}( {\omega\; t} )}{\sin\mspace{11mu}( {\omega\; t} )}} )}} \\{= {\tan^{- 1}( {- \frac{\sin\mspace{11mu}( {{\omega\; t} + \frac{\pi}{2}} )}{\sin\mspace{11mu}( {\omega\; t} )}} )}} \\{= {\tan^{- 1}( \frac{\sin\mspace{11mu}( {{\omega\; t} + \frac{\pi}{2}} )}{\sin\mspace{11mu}( {{\omega\; t} + \pi} )} )}} \\{= {\tan^{- 1}( \frac{\sin\mspace{11mu}( {{\omega\; t} + \frac{\pi}{2}} )}{\cos\mspace{11mu}( {{\omega\; t} + \pi - \frac{\pi}{2}} )} )}} \\{= {{\omega\; t} + \frac{\pi}{2}}} \\{= \psi}\end{matrix}\begin{matrix}{{a_{ang}(t)} = {\tan^{- 1}( \frac{a_{x}(t)}{a_{y}(t)} )}} \\{= {\tan^{- 1}( \frac{{- \sin}\mspace{11mu}( {\omega\; t} )}{{- \cos}\mspace{14mu}( {\omega\; t} )} )}} \\{= {\tan^{- 1}( \frac{\sin\mspace{11mu}( {{\omega\; t} + \pi} )}{\cos\mspace{11mu}( {{\omega\; t} + \pi} )} )}} \\{= {{\omega\; t} + \pi}} \\{= {\psi + \frac{\pi}{2}}}\end{matrix}$Which corresponds to constant expected values on the IMU outputs:a _(u) _(tr) (t)=ω² ra _(v) _(tr) (t)=0v _(ψ) _(tr) (t)=ω

With regard to IMU errors, there are a nearly endless number ofpotential error sources in an IMU system, but the following discussionsimplifies the problem down to independent Gaussian noise parameters foru and v. ψ is assumed to be without error.

$\begin{matrix}{{a_{u_{i}}(t)} = a_{{u_{tr}{(t)}} + {n_{u_{i}}{(t)}}}} \\{{a_{v_{i}}(t)} = a_{{v_{tr}{(t)}} + {n_{v_{i}}{(t)}}}} \\{= {n_{v_{i}}(t)}}\end{matrix}$ $\begin{matrix}{{v_{\psi_{i}}(t)} = v_{{\psi_{tr}{(t)}} + {n_{\psi_{i}}{(t)}}}} \\{= {v_{\psi_{tr}}(t)}} \\{{\psi_{i}(t)} = {{\omega\; t} + \frac{\pi}{2}}}\end{matrix}$In this case, v_(ψ) is meant to indicate angular rate or dψ/dt. The nvalues are noise. Additional errors stem from discretely sampling thesignal in time, as discussed below. There may also be bias terms foreach sensor, which are assumed to be zero here for simplicity.

Dead reckoning implements the reverse of the derivation above. Based onan initial position and sensor measurements, the system constructs a newposition. The process can then be repeated using the newly constructedposition as the initial position for the next point. Acceleration inIMU-space is decomposed into Cartesian world-space coordinate as:a _(x) _(i) (t)=a _(u) _(i) (t)·cos(ψ_(i)(t))a _(y) _(i) (t)=a _(v) _(i) (t)·sin(ψ_(i)(t))Instead of doing a full continuous integral, an IMU is typically sampledat a given sampling rate, f_(s). Velocity is the summation of prioraccelerations times the sample period:

${v_{x_{i}}( t_{1} )} = {{v_{x_{i}}( t_{0} )} + {\underset{k = 0}{\sum\limits^{\int{s{({t_{1} - t_{0}})}}}}{\frac{1}{\int s}{a_{x_{i}}( {t_{0} + {k\frac{1}{\int s}}} )}}}}$${v_{y_{i}}( t_{1} )} = {{v_{t_{i}}( t_{0} )} + {\underset{k = 0}{\sum\limits^{\int{s{({t_{1} - t_{0}})}}}}{\frac{1}{\int s}{a_{t_{i}}( {t_{0} + {k\frac{1}{\int s}}} )}}}}$And position is the summation of prior velocities:

${s_{x_{i}}( t_{1} )} = {{s_{x_{i}}( t_{0} )} + {\underset{k = 0}{\sum\limits^{\int{s{({t_{1} - t_{0}})}}}}{\frac{1}{\int s}{v_{x_{i}}( {t_{0} + {k\frac{1}{\int s}}} )}}}}$${s_{y_{i}}( t_{1} )} = {{s_{y_{i}}( t_{0} )} + {\underset{k = 0}{\sum\limits^{\int{s{({t_{1} - t_{0}})}}}}{\frac{1}{\int s}{v_{y_{i}}( {t_{0} + {k\frac{1}{\int s}}} )}}}}$

With regard to errors due to sampling, even without noise on theaccelerometer signal, there will be some error in the dead-reckonedresult due to the discrete time sampling of acceleration. For an IMUtraveling a circular path, the acceleration vector is changingcontinuously, but the system samples that acceleration periodically andassumes the value to be constant throughout the sample period. Moresophisticated filtering techniques can be used to mitigate this to someextent, depending on the acceleration profile, but one useful techniqueis to employ the naive approach. FIG. 2 illustrates a graph 200 showingan IMU position track at two sample rates. In the graph 200, the dashedcircle 210, which is centered at (0,0) with a radius of 200 meters,indicates the true path of the IMU on the moving body. The thicker linepath 220 shows the error over the course of 6000 seconds when sampled at1 kHz, and the thinner line path 230 shows the significantly greatererror when sampled at 100 Hz. In the static test case, this is not anissue because the acceleration remains constant (and zero) throughoutthe sample period.

With this in mind, one can consider random walk growth in velocity errorby looking at the noise on the accelerometer. FIG. 3 provides a graph300 of a static case/use of a single axis accelerometer. The graph 300shows simulated accelerometer values with a mean of zero and a standarddeviation of 0.001 g. Tracing the noise through the dead reckoningcalculations as shown with graph 400 of FIG. 4 , the accelerometervalues are integrated into velocity values which exhibit a random walkwith a non-zero mean, whereas truth would be a velocity of zero. Thevelocity then integrates into a position as shown with graph 500 of FIG.5 , with significant error growth.

The same pattern occurs in the circular test case. FIG. 6 provides agraph or plot 600 of x acceleration, shown to give an indication of themagnitude of the noise relative to the signal, and FIG. 7 provides agraph or plot 700 showing accelerometer error. This can be integratedinto a velocity as shown with graph or plot 800 of FIG. 8 , and thevelocity error, shown in plot or graph 900 of FIG. 9 , also contains anoscillation due to the sampling of the signal. FIG. 10 is a graph 1000showing the full 2D position track for the circular test case, and FIG.11 is a graph 1100 of the error from truth over time for the track ofFIG. 10 .

At this point in the discussion, it may be useful to explain operationsof a dead reckoning module (such as module 174) to provide deadreckoning with a computer vision-based positioning/navigation system(such as system 170 in FIG. 1 ). There are multiple methods of deadreckoning using computer vision, using either 3D or 2D inputs, that maybe utilized to implement the present invention and its navigationsystems. The following provides two useful methods that use 2D images,and one skilled in the art will recognize that the descriptions are lessformal than those provided above because full derivation, althoughsignificantly more complex and varying with implementation, is wellunderstood in the field of vision-based navigation and relatedprocessing of images.

One dead reckoning technique involves optical flow. Optical flowmeasures movement in the light and dark regions of an image, and itassumes that such movement implies motion in what is being imaged. Byadding the assumptions that what is being imaged is rigid and planar,one can simplify the calculations. The optical flow mechanism can beexpressed visually as shown in FIG. 12 with first and second images 1210and 1220 (e.g., a frame pair) from a handheld camera. The dead reckoningmodule may determine how areas of light and dark have moved from theirposition in image 1210 to their position in image 1220 (e.g., theselight and dark areas correspond to the color or gray intensities of thepixels). If it assumed that the imaged road surface has remained static,the module can determine the camera motion. The diagram of FIG. 12 callsout the changes in x, y, and ψ. The scale of the second image 1220 waschanged, which corresponds to a change in z.

Another technique for providing dead reckoning with a CV system issimultaneous location and mapping (SLAM), and, as the name implies, SLAMdoes two things at the same time. It maps points in the surroundingenvironment and determines the (typically relative) location of thecamera.

With regard to a need for overlapping imagery, all image-based deadreckoning methods call for the imagery being used to include enoughcommon scenery to support common measurements and alignment betweenimages. If the camera moves enough between frames in a frame pair thatthere is no common scenery/features, then dead reckoning is notpossible. In order to maintain positioning in real time, the CV systemshould have enough computation power to complete processing of eachframe, along with the earlier data collected, before the next frame (orframe pair) is captured.

The use of multiple cameras (such as cameras 176 of FIG. 1 ) in a CVpositioning system can be desirable for providing a variety of benefits.First, overlapping imagery from cameras with known relative orientationand baseline can allow direct estimates of scale (e.g., the size ofthings in the image). This is helpful in converting image-basedmeasurements to units compatible with inertial and other sensors and ininterpreting the location of points in 3D space. Second, wider, ormultiple, fields of view are provided by multiple cameras that can lendgeometric strength to the motion estimates. Third, using multiplecameras can allow the CV system to implement staggered shutter times,which can enable shorter time periods between captures.

With regard to correcting inertial drive using the output of a CVsystem, the inventive navigation methods taught herein involve using theCV system measurements or outputs to help contain the inertial drift.This is achieved in a way that does not require enough computation powerto permit continuous overlap of imagery. In the new navigation method,two images (or a frame pair) are captured that do overlap. The deadreckoning module of the CV system then takes as long as needed toprocess the pair of images while the inertial navigator (e.g., inertialnavigation unit 150 in FIG. 1 ) dead reckons from the last known point.When processing of the images is complete, the CV system provides anestimated change of pose between these two images, which, combined withthe time interval between the capture times of the two images, providesa pose rate or velocity. This known velocity is then used (e.g., by theintegration module 144 or its filter mechanism 148 of the navigationsystem 130 of FIG. 1 ) to remove the error from the inertial navigatorvelocity estimate with the caveat that this error is calculated at thetime that the imagery was captured and not at the time that thecomputation is complete.

There are many ways to design an inertial navigator. For illustrationpurposes, a simple integrator may be used and rotations can be ignoredto demonstrate the advantage of the combined solution taught herein fora new navigation system (i.e., one that combines an IMU with a CVsystem). The velocity estimates from the inertial system can beconsidered at a sequence of epochs, with each epoch including severalmeasurement periods of the accelerometer:

At  t₀ : v₀ = 0${{At}\mspace{14mu}{t_{1}:v_{1}}} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{1}}{a(t)}} + {\overset{\sim}{v}}_{1}}$${{At}\mspace{14mu}{t_{2}:v_{2}}} = {{v_{1} + {\sum\limits_{t = t_{1}}^{t_{2}}{a(t)}} + {\overset{\sim}{v}}_{2}} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{2}}{a(t)}} + {\overset{\sim}{v}}_{1} + {\overset{\sim}{v}}_{2}}}$${{At}\mspace{14mu}{t_{3}:v_{3}}} = {{v_{2} + {\sum\limits_{t = t_{2}}^{t_{3}}{a(t)}} + {\overset{\sim}{v}}_{3}} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{3}}{a(t)}} + {\overset{\sim}{v}}_{1} + {\overset{\sim}{v}}_{2} + {\overset{\sim}{v}}_{3}}}$and so on, where the v's are the true velocities, the a's areaccelerations as measured at time t, and the {tilde over (v)}'s are thevelocity errors at that time (from whatever source).

At each epoch t_(n), a frame pair is captured and the computation of theCV estimate of motion is completed at t_(m+1), giving an independentestimate of velocity and an error endemic to the CV calculation{circumflex over (v)}. This leads to a new chain of calculations:

At t_(o):v ₀=0

At t_(i) before the CV calculation:

$v_{1} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{1}}{a(t)}} + {\overset{\sim}{v}}_{1}}$

At t_(i) after the CV calculation:

$v_{1} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{1}}{a(t)}} + \overset{\sim}{v} + {\overset{\sim}{v}}_{1}}$

At t_(o) before the CV calculation:

$v_{2} = {{v_{1} + {\sum\limits_{t = t_{1}}^{t_{2}}{a(t)}} + {\overset{\sim}{v}}_{2}} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{2}}{a(t)}} + \overset{\sim}{v} + {\overset{\sim}{v}}_{1} + {\overset{\sim}{v}}_{2}}}$

At t_(o) after the CV calculation:

$v_{2} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{2}}{a(t)}} + \overset{\sim}{v} + {\overset{\sim}{v}}_{2}}$

At t_(s) before the CV calculation:

$v_{3} = {{v_{2} + {\sum\limits_{t = t_{2}}^{t_{2}}{a(t)}} + {\overset{\sim}{v}}_{3}} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{3}}{a(t)}} + \overset{\sim}{v} + {\overset{\sim}{v}}_{2} + {\overset{\sim}{v}}_{3}}}$

At t_(s) after the CV calculation:

$v_{3} = {v_{0} + {\sum\limits_{t = t_{0}}^{t_{2}}{a(t)}} + \overset{\sim}{v} + {\overset{\sim}{v}}_{3}}$Hence, it can be seen that, while inertial based velocity errors grewlinearly with time, the CV corrected velocity error is now bounded.FIGS. 13 and 14 are graphs 1300 and 1400 showing full 2D positiontracking with when correction of inertial drift is provided via a CVsystem for the static and circular motion simulations, respectively, andthe graphs 1300 and 1400 illustrate the benefit of the drift correctionfrom the CV system (or integration of the IMU and CV systems) in similarsimulations as discussed above.

FIGS. 15A and 15B illustrate with two timelines 1510 and 1520 the timingof vision-based positioning system updates to an IMU system. In thetimelines 1510 and 1520, capture of images or frames with one (or more)cameras of a vision-based positioning system are shown. Navigationupdates are provided as shown at 1514 and 1524 by a vision-basedpositioning system. The navigation system uses output of an IMU toprovide navigation between video navigation updates 1514 and 1524. Onthe first timeline 1510, the frame rate for capture of frames 1512 isten times (10×) the video navigation update rate (or the videonavigation rate is 0.1 times the frame rate). As shown with the secondtimeline 1520, more frequent video updates are possible when aided bythe IMU unit. In this example, the frame rate is six times (6×) thevideo navigation update rate (or the video navigation rate is 0.17 timesthe frame rate).

More specifically, in one example, an IMU may be used to sense adirection of a field of view movement, and a vision-based navigationsystem may use this heading or direction of travel information in theIMU navigation data to select which portion of image or frame pairs toprocess. This can be seen in FIG. 16 with first-captured image 1610 andsecond-captured (later-captured) image 1620. An IMU of a navigationsystem may be used to sense or determine a heading/direction of travel1622 that indicates field of view movement toward the right from image1610 to image 1620. A video navigation system may use this headinginformation to determine it should devote computational resources to theleft side of the images 1610 and 1620 as information from the area 1630will disappear (from later-captured images) first.

FIG. 17 shows another two captured images (or frame pair) to beprocessed by a navigation system of the present description. The IMUnavigation data or output helps disambiguation of video data by the CVsystem (or video unit). Particularly, the IMU output may provide aheading 1715, and the CV system can use this information to determinematching features/elements in the images 1710, 1720 (to then determinepose rate/velocity). As shown, the images 1710 and 1720 include a numberof lampposts, and the heading/DOT 1715 from the IMU navigation unitallows the CV system to identify matching lampposts in the images 1710and 1720 such as posts 1712 and 1716 in each image/frame 1710 and 1720.

FIG. 18 shows another set of sequentially captured frames/images 1810,1820, and 1830 that are to be processed by a navigation system of thepresent description. An IMU can be used on a moving body carrying thecamera capturing images 1810, 1820, and 1830 to sense field of viewmovement (or heading/DOT) as shown with arrows 1805. The vision-basedpositioning system (or video unit) can use this data from the IMU toanticipate where new information in each subsequent frame (frame 1820after frame 1810 and frame 1830 after frame 1820) will appear. In thisexample, more of the house is visible in frame 1820 in the lower rightcorner than in the frame 1810, and a fire hydrant appears in frame 1830also in the lower right hand corner that was not present in frames 1810and 1820. This can be predicted based on the heading/DOT 1805 providedin the IMU output/navigation data.

In some embodiments, the navigation system further is configured toinclude software to perform post-processing. In the post-processingmode, forward/backward smoothing can be applied. The processing periodcan be allowed to vary with the complexity of the calculation and theavailable processor resources. Simple extensions allow specialconsideration of the static (or near static) case by detecting that theframes captured for processing period t+1 match the frames fromprocessing period t, which further reduces the drift of the videosolution. IMU measurements can be fed into the VN unit or system (alsolabeled CV system and video-based positioning/navigation system) toreduce the computation effort required to match frames. The algorithmcan be further extended to recognize loop closures including recognizingthat the system has returned to a starting point.

The navigation systems may implement a variety of vehicular cameraarchitectures to implement the ideas taught herein includingvision-based navigation integrated with inertial navigation. Vehiclespresent a unique environment for vision-based navigation. The speeds ofmovement, the proximity to other vehicles, the mass of the vehicle, andthe environments the vehicle positions itself within are all quitedifferent from applications such as drones or handheld devices.

With regard to scene selection, there are several choices of scene, asfar as what the camera is looking at. SLAM, for example, has benefitssuch as the ability to close loops by recognizing scenes we've alreadyseen. With regard to surface tracking, one characteristic of being in apassenger vehicle is that there is typically a relatively smooth roadbeneath it that is suitable for surface tracking techniques similar, butnot limited, to those described in U.S. Pat. No. 8,229,166, which isincorporated herein in its entirety by reference. Looking down at theroad provides several key advantages including: (a) predictability asthe road will always be there; (b) correlated motion because there isless opportunities to be confused by objects in the scene moving in waysthat do not reflect the motion of the camera (and vehicle it is attachedto); and (c) 2D surfaces give features that do not change appearancewith perspective.

With regard to camera placement on a vehicle body, among the interestinglocations to place the navigation cameras would be: (a) rear viewmirror: providing a view forward and protected by the windshield whichwill typically be maintained by the driver to establish a clear view;(b) rear bumper/trunk/license plate area: gives a clear and more openview of the ground and is in a position to see more road surface betweenthe tracked vehicle and surrounding traffic; (c) side-view mirrors asthis gives a clear view down and out to both sides; and (d) under thevehicle, which gives a predicable view with more predictable lighting(or lack thereof).

With regard to camera count in a navigation system, a camera-based (orvision-based) navigation system can benefit in different ways fromincorporating multiple cameras. All multiple camera systems can benefitfrom the redundancy of multiple sensors. When using multiple overlappingcameras, overlapping views can give the system a direct estimate ofscale in the imagery that is captured. By knowing the cameraorientations and relative locations, the precise size of objects in theview can be determined by the system (e.g., by its dead reckoning moduleor other software). Alternatively, or in addition, multiple overlappingviews can help mitigate rolling shutter effects if the sensors arearranged in a way that the rolling shutter affects the image differentlyin each sensor. For example, if one camera tends to shear the image tothe left and the other is mounted so that the rolling shutter if rotated180 degrees and shears the image to the right, then the system candetermine the correct un-sheared image. At the same time, if themagnitude of the image shear is determined, then it can serve as ameasurement of the relative motion between the camera and the scene.

In some embodiments of a navigation system, it may be useful toimplement multiple non-overlapping cameras for the benefits it provides.These benefits include: (a) more cameras increase the likelihood that atleast one has a view of a trackable scene; (b) more cameras lessen thechance that all cameras are equally effected by sunstrike or otherexposure challenges; (c) a wider field of view gives more geometricstrength to the positioning solution; and (d) some vision-basedpositioning techniques give greater observability to certain degrees offreedom in camera motion. With regard to this last benefit, for example,a surface tracking camera may be strongest when tracking motionperpendicular to the camera axis and rotation around the camera axis,and it may be weaker in the other dimensions. Having an additionalcamera mounted to align its strong dimensions with the weaker dimensionsof the first camera will improve the strength of the overall system. Ifone camera is looking toward the front of the vehicle and a secondcamera is looking toward the back, the system can track the motion ofscenes from the front camera as they reappear in the rear camera toextend the continuity of tracking.

With regard to optical design of the navigation system, it may be usefulto utilize tilted optics. Tilting the lens axis relative to the plane ofthe sensor can shift the focal plane according to the Scheimpflugprinciple. Such a design, when mounted on a vehicle with a predicableattitude relative to the surface of the road, would allow the roadsurface to be kept in focus from the vehicle outward rather than havingthe focal place intersect the road surface at a particular distance andcausing the surface to be out of focus nearer to and further from thatline of intersection.

With regard to artificial illumination in the navigation system,lighting often will be unpredictable in vehicle applications due to timeof day, weather, overpasses, tunnels, and the like. The ability toartificially illuminate may be provided on the vehicle in the navigationsystem as such lighting of the scene extends the useful working envelopeof the system. Lighting may be designed to take into account the vehicledrivers comfort and safety and that of other drivers on the road andpedestrians nearby. Infrared (IR) lighting would allow illumination thatthe image sensor could detect but that would not be visible to humans,reducing distraction.

In some cases, it may be useful to include RGB IR lighting in theartificial illumination assembly of the navigation system. Silicon andother image sensors are naturally sensitive to infrared light, but it istypically filtered out in photographic cameras. For cameras dedicated tonavigation, this filter could be removed to increase sensitivity to IR.For monochrome cameras, the entire sensor could be sensitive. Monochromesensors have the additional advantage of not filtering out specificcolors per pixel therefore accepting more light into the pixel allowingoperation in lower ambient light. For color cameras, which typicallyhave individual color filters over pixels in a pattern similar to theBayer pattern or some other known method, one or more of the colorfilters could be extended to permit IR to pass through—for example thered (or some other) pixel filters could be widened to accept infrared inaddition to red light. Photographic image sensors often have twice asmany green pixels as they do red and blue because the human eye tends tobe more sensitive to green light. Half of those green pixel filterscould be removed and left unfiltered to permit more light of all colorsincluding infrared into the pixel.

With regard to combinations with depth cameras, so called “depthcameras” may be included in the navigation system as they provideillumination to the scene in such a way that the camera can determine adistance to each pixel. This may be a structured light pattern that isused to resolve depth, or a strobe intended for measurement of time offlight. If an RGB sensor leaves one of 4 pixels (or some other ratio)unfiltered, the sensor can integrate IR depth sensing into the RGBcamera. In addition to the other benefits of global shutter, it alsopermits synchronization with short but intense pulse lighting making iteasier to use depth cameras in varying ambient light.

With regard to maintenance of the navigation system, a camera used fornavigation can be adapted or designed to be able to detect when its viewhas been obscured by dust, dirt, water, or lens damage and, in response,to notify the driver, vehicle safety systems, or both. Likewise, thecamera may be adapted or designed to be able to detect if it is beingblocked by a towed trailer or other object and, in response, to providecorresponding notification to the driver, vehicle safety systems, orboth.

The rolling shutter compensation module 142 of the navigation system 130of FIG. 1 may be implemented with a number of useful functions tocorrect problems associated with use of image sensors 176 using rollingshutter image capture to capture the frame pairs 182. For certaingeometries, under certain dynamics, and with certain external aidinginformation, the lost information (due to rolling shutter image capture)can be recovered by the module 142.

With the camera looking at a smooth continuous surface, there is less 3Dinformation to be processed. The depth to a point can often bedetermined as a simple function of the other two dimensions. Additionalsensors such as accelerometers, gyroscopes, speedometers (such asspeedometer 124) and wheel rotation sensors can help the system (e.g.,system 130 with rolling shutter compensation module 142) determine howthe camera (such as camera 176) is moving as each row is captured.Typically, the limitation on using such sensors is that they do not havesufficient time resolution to model the camera motion with sufficientaccuracy relative to the row to row timings. In a handheld application,for example, it is possible to induce enough change of acceleration tothe device that interpolating accelerometer data will not accuratelymodel the true acceleration value between measurements. This, in turnleads to errors in velocity estimates, which leads to errors in camerapose estimates at each row capture time.

Under certain dynamic constraints, the change in camera pose between rowcaptures can more easily be recovered. When the camera is attached to alarge mass, such as a vehicle body 110, for example, the system willresist rapid changes in acceleration. Under these conditions, suchinterpolations of sensor data will become much more reliable.

As an example of how such rolling shutter effects might be corrected(e.g., by module 142), consider a vehicle mounted camera looking at theroad surface in front of the vehicle. Assume that a camera has a rollingshutter with the first row imaged furthest from the vehicle and the lastrow imaged closest to the vehicle. If the vehicle is stationary, theimage will accurately represent the road surface in its field of view.If the vehicle begins rolling forward, then the image will start tobecome compressed in the direction along the road because as each row iscaptured the vehicle will have moved forward some amount meaning thatthe second row will be measured further along the row than it would havebeen if the vehicle were stationary. If the vehicle is moving rapidlyenough, the image can become inverted because the second row may becaptured far enough down the road that it is imaging a region beyondwhere the first row is captured.

If provided with a speedometer and accelerometer input, for example,this distortion could be reversed (by the module 142, for example)because we now know the rate of movement in the vehicle and thus thespacing between row captures. The mass of the vehicle better enforcesthe condition that the rate of change in acceleration is small relativeto the image capture time. With regard to multiple sensors (such ascameras 176 in navigation system 130 of FIG. 1 ), if two image sensorsare mounted in such a way that the motion of the scene relative to thecamera affects each image differently, the rolling shutter effect can bebetter compensated for and the result can yield additional information.For example if two cameras are looking at the same scene, but one isrotated around the camera axis by 180°, then the shearing effect in eachimage will be in opposite directions. If one sensor shears left, forexample, and the other shears right, then each can be compensated backto the correct non-sheared image. In addition, once the magnitude of themotion related shear is determined, it can be used as an estimate of therelative velocities between the camera and scene.

As can be seen by the preceding description, the inventors areaddressing problems arising solely due to the recent development ofcomputer-based navigation systems for automobiles (and othervehicles/moving bodies). Particularly, computer vision-based navigationsystems have issues including computation-intensive processing androlling shutter distortion that limited their use in many applicationsincluding navigation that requires more real time input. With theseissues in mind, the inventors created a navigation system that functionsmuch more efficiently and/or quickly to produce navigation data in realtime. This navigation data may include a present position of a vehiclein a space and its current heading, and the navigation data may be inputto the vehicle's drive system for display on one or more GUIs and/or foruse in operating the drive system to navigate through the space.

While this disclosure contains many specifics, these should not beconstrued as limitations on the scope of the disclosure or of what maybe claimed, but rather as descriptions of features specific toparticular embodiments of the disclosure. Furthermore, certain featuresthat are described in this specification in the context of separateembodiments can also be implemented in combination in a singleembodiment. Conversely, various features that are described in thecontext of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and/or parallelprocessing may be advantageous. Moreover, the separation of varioussystem components in the embodiments described above should not beunderstood as requiring such separation in all embodiments, and itshould be understood that the described program components and systemscan generally be integrated together in a single software and/orhardware product or packaged into multiple software and/or hardwareproducts.

The above described embodiments including the preferred embodiment andthe best mode of the invention known to the inventor at the time offiling are given by illustrative examples only.

We claim:
 1. A navigation system for vehicles and other moving bodies,comprising: an inertial navigation unit including an inertialmeasurement unit (IMU) and an IMU dead reckoning module processing a setof measurements provided by the IMU to generate a first output, whereinthe navigation system is positioned on a navigable body with a drivesystem, wherein the navigation system generates navigation data for useby the drive system to move the navigable body through a space, andwherein the first output includes a first velocity and a heading of thenavigable body; a vision-based navigation system including an imagesensor sequentially capturing a pair of images of the space and furtherincluding a vision-based dead reckoning module processing the pair ofimages based on the heading to generate a second output including asecond velocity of the navigable body; and a processor with anintegration module processing the first and second outputs to generatethe navigation data, wherein the navigation data includes the firstvelocity or a first position calculated based on the first velocityduring a time period while the vision-based navigation system isgenerating the second output at the end of which the first velocity ismodified based on the second velocity.
 2. The navigation system of claim1, wherein the vision-based dead reckoning module processes the pair ofimages based on the heading to process portions of the pair of imageswith information predicted to be lost with further travel of thenavigable body or to predict portions of later frames with newinformation.
 3. The navigation system of claim 1, wherein, after thefirst velocity is modified based on the second velocity, the IMU deadreckoning module processes an additional set of measurements provided byIMU to generate a third output including a third velocity of thenavigable body and wherein the integration module processes the thirdoutput to generate the navigation data based on the third output and themodified first velocity.
 4. The navigation system of claim 1, whereinthe vision-based navigation system provides the second output atnon-uniform time intervals as determined by computational loading. 5.The navigation system of claim 1, wherein the image sensor comprises atleast two video cameras and wherein at least a pair of the video camerasare directed at an angular offset of 180 degrees.
 6. The navigationsystem of claim 1, wherein the image sensor is a rolling shutter imagesensor and wherein the vision-based navigation system further comprisesa rolling shutter compensation module processing the pair of imagesbased on a speed of the navigable body to reverse distortion caused by arolling shutter effect.
 7. The navigation system of claim 1, wherein thepair of images include a region of overlap.
 8. A navigation system forvehicles and other moving bodies, comprising: an inertial navigationunit including an inertial measurement unit (IMU) and an IMU deadreckoning module processing a set of measurements provided by the IMU togenerate a first output, wherein the navigation system is positioned ona navigable body and wherein the first output includes a first velocityand a heading of the navigable body; a vision-based navigation systemincluding an image sensor sequentially capturing a pair of images of aspace and further including a vision-based dead reckoning moduleprocessing the pair of images based on the heading to generate a secondoutput including a second velocity of the navigable body; and aprocessor with an integration module processing the first and secondoutputs to generate navigation data including modifying the firstvelocity based on the second velocity, wherein, after the first velocityis modified based on the second velocity, the IMU dead reckoning moduleprocesses an additional set of measurements provided by the IMU togenerate a third output including a third velocity of the navigable bodyand wherein the integration module processes the third output togenerate the navigation data based on the third output and the modifiedfirst velocity.
 9. The navigation system of claim 8, wherein thenavigation data includes the first velocity or a first positioncalculated based on the first velocity during a time period while thevision-based navigation system is generating the second output at theend of which the first velocity is modified based on the secondvelocity.
 10. The navigation system of claim 8, wherein the vision-baseddead reckoning module processes the pair of images based on the headingto process portions of the pair of images with information predicted tobe lost with further travel of the navigable body or to predict portionsof later frames with new information.
 11. The navigation system of claim8, wherein the vision-based navigation system provides the second outputat non-uniform time intervals as determined by computational loading.12. The navigation system of claim 8, wherein the image sensor comprisesat least two video cameras and wherein at least a pair of the videocameras are directed at an angular offset of 180 degrees.
 13. Thenavigation system of claim 8, wherein the image sensor is a rollingshutter image sensor and wherein the vision-based navigation systemfurther comprises a rolling shutter compensation module processing thepair of images based on a speed of the navigable body to reversedistortion caused by a rolling shutter effect.
 14. The navigation systemof claim 8, wherein the pair of images include a region of overlap. 15.A navigation method for vehicles and other moving bodies, comprising:processing a set of measurements, provided by an IMU on a navigablebody, to generate a first output including a first velocity and aheading of the navigable body; sequentially capturing a pair of imagesof a space in which the navigable body is traveling; processing the pairof images based on the heading to generate a second output including asecond velocity of the navigable body; and processing the first andsecond outputs to generate navigation data for the navigable body,wherein the navigation data includes the first velocity or a firstposition calculated based on the first velocity during the generating ofthe second output and wherein, after the generating of the secondoutput, the first velocity is modified based on the second velocity. 16.The navigation method of claim 15, wherein the method further comprisesprocessing the pair of images based on the heading to process portionsof the pair of images with information predicted to be lost with furthertravel of the navigable body or to predict portions of later frames withnew information.
 17. The navigation method of claim 15, furthercomprising, after the first velocity is modified based on the secondvelocity, processing an additional set of measurements provided by theIMU to generate a third output including a third velocity of thenavigable body and then processing the third output to generate thenavigation data based on the third output and the modified firstvelocity.
 18. The navigation method of claim 15, wherein the sequentialcapturing is performed by at least two video cameras directed at anangular offset of 180 degrees.
 19. The navigation method of claim 15,wherein the sequential capturing is performed by a rolling shutter imagesensor and the method further includes processing the pair of imagesbased on a speed of the navigable body to reverse distortion caused by arolling shutter effect.